Evaluating a domain-specialist-oriented knowledge management system

We discuss the evaluation of a tool designed to allow domain specialists to manage their own knowledge base. We present the evaluation as a two-phase process: In the first phase we assess whether the tool has met its objectives of allowing those not trained in logical formalisms to effectively represent and manipulate knowledge in a computer. By studying use of the tool by its intended users, we conclude that it has met this objective. In the second phase of the evaluation, we assess what aspects of the tool have in fact led to its success. To do this we study what tasks are performed by users, and what features of both knowledge representation and user interface are exercised. We find that features for manipulating the inheritance hierarchy and naming concepts are considered the most valuable. Our overall conclusion is that tool research must involve this two-phase approach if the others are to learn from the work – the research has much less value unless it can be determined which features should most profitably be adopted by others.